chore: start Xetification - #215

huggingface_hub/src/huggingface_hub/file_download.py at main · huggingface/huggingface_hub · GitHub
Initial call to hf_hub_download

huggingface_hub/src/huggingface_hub/file_download.py at main · huggingface/huggingface_hub · GitHub
Call to _hf_hub_download_to_local_dir

huggingface_hub/src/huggingface_hub/file_download.py at main · huggingface/huggingface_hub · GitHub
Fetch metadata from HF, metadata contains XetFileData:

@dataclass(frozen=True)
class XetFileData:
    file_hash: str
    refresh_route: str

First action item: successfully fetch XetFileData from the hub and enrich our current Metadata struct in Swift

Call _download_to_tmp_and_move with our newly sourced metadata:
huggingface_hub/src/huggingface_hub/file_download.py at main · huggingface/huggingface_hub · GitHub

Call into xet_get:
huggingface_hub/src/huggingface_hub/file_download.py at main · huggingface/huggingface_hub · GitHub

xet_get imports from hf_xet which is the actual Rust + PyO3 crate.

The key function for this is download_files, which is actually Rust code:

huggingface_hub/src/huggingface_hub/file_download.py at main · huggingface/huggingface_hub · GitHub

If we can expose download_files from Rust via C to Swift, we are golden! The rest is just Swift enrichment.

Successfully downloaded via Xet in the following mini package: https://github.com/FL33TW00D/xet-swift/blob/master/xet-swift/Tests/xet-swiftTests/xet_swiftTests.swift

Just needs to be integrated into the Hub package as we are away!

@pcuenca probably ready for you to look over! just adds the Xet metadata to our struct, doesn't do anything with it yet!